DBP: Support new dataSource property #2267

Bunn · 2024-02-26T16:52:09Z

Task/Issue URL: https://app.asana.com/0/1204167627774280/1206602738361283/f

Description:
Add new datasource property support

Steps to test this PR:
(Use C-S-S from main to test this)

Do a full scan/opt-out on an old broker with old JSON, see if it works
Change the JSON (you can use DBP debug mode to make it easier) and add a datasource to an action, like:

"dataSource": "extractedProfile",
"dataSource": "userProfile",

ex:

          "actionType": "navigate",
          "id": "cf670748-0509-4a78-8b46-d8cf7135538e",
          "dataSource": "extractedProfile",

Check if we're correctly sending the dataSource property back to CCF
You can do this by breakpointing here

macos-browser/LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/CCF/DataBrokerProtectionFeature.swift

Line 127 in 55375b3

    
           func pushAction(method: CCFSubscribeActionName, webView: WKWebView, params: Encodable) {

Internal references:

Pull Request Review Checklist
Software Engineering Expectations
Technical Design Template
Pull Request Documentation

# Conflicts: # DuckDuckGo.xcodeproj/project.pbxproj # DuckDuckGo.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved # LocalPackages/DataBrokerProtection/Package.swift # LocalPackages/LoginItems/Package.swift # LocalPackages/NetworkProtectionMac/Package.swift # LocalPackages/PixelKit/Package.swift # LocalPackages/SubscriptionUI/Package.swift # LocalPackages/SwiftUIExtensions/Package.swift # LocalPackages/SyncUI/Package.swift # LocalPackages/SystemExtensionManager/Package.swift # LocalPackages/XPCHelper/Package.swift

github-actions · 2024-02-28T16:48:25Z

	Warnings
⚠️	PR has more than 500 lines of code changing. Consider splitting into smaller PRs if possible.

Generated by 🚫 dangerJS against f125094

jotaemepereira

@Bunn I have some questions about some broker JSON files

jotaemepereira · 2024-02-28T19:07:20Z

...okerProtection/Sources/DataBrokerProtection/Resources/JSON/advancedbackgroundchecks.com.json

@@ -45,7 +45,9 @@
              "separator": ","
            },
            "profileUrl": {
-              "selector": ".link-to-details"
+              "selector": ".link-to-details",


⚠️ Here, we added the identifier, but we didn’t bump the version on this file. What will happen to users who find this? Or we didn’t release the last change for this?

You're right. I think this needs to be bumped to 0.1.6.
@brianhall, wdyt?

This whole versioning thing was inherited and needs to be defined a little better, I'll work on it. 😄 Technically the identifier isn't visible to any client, so it won't matter if they see it, but happy for this to get bumped to 0.1.6, I'll add a commit with all changes based on the feedback in a minute.

@brianhall Let me change the wording. What I meant with “users who find this”, I meant that if a user has this extract JSON broker version on their computers, they will not get this identifier change, because for them it will be the same version.

@brianhall, I believe it matters because we only parse/save the new JSON if there's a version change. The JSON is not always read from the file; we store it serialized in the DB. Therefore, if we don't bump the version here, we won't send the identifier to C-S-S.

Ah, thanks both! That's useful context, understood.

I'll add a commit with all changes based on the feedback in a minute.

Just to confirm, are you doing a commit on this branch or in the json repo for me to fetch the changes here?

Oops, sorry. Just bumped a commit here and will reflect back in the broker json repo.

jotaemepereira · 2024-02-28T19:08:48Z

...s/DataBrokerProtection/Sources/DataBrokerProtection/Resources/JSON/fastpeoplesearch.com.json

+{
+  "name": "FastPeopleSearch",
+  "url": "fastpeoplesearch.com",
+  "version": "0.1.5",


❓ This is weird; why is this a new broker, but the version is 0.1.5? 🤔

I believe this is simply due to the nature of how the generator works; it sets the same version for all of them. I don't see it as an issue

Yep, that's right @Bunn. I think we need to decide whether the versioning is "a versioned release of one or more changes" (the mode I've been operating under) or if each json file gets its own individual version. I've also been operating with the former mental model.

jotaemepereira · 2024-02-28T19:09:29Z

...lPackages/DataBrokerProtection/Sources/DataBrokerProtection/Resources/JSON/inforver.com.json

@@ -1,7 +1,7 @@
 {
  "name": "Inforver",
  "url": "inforver.com",
-  "version": "0.1.4",
+  "version": "0.1.5",


⚠️ There was no change in Verecor, and I do not see the identifier. Should it be here?

Inforver now redirects to persontrust.com, so I didn't bother updating the identifier, it's ok as-is.

jotaemepereira · 2024-02-28T19:10:14Z

...ckages/DataBrokerProtection/Sources/DataBrokerProtection/Resources/JSON/officialusa.com.json

@@ -1,7 +1,7 @@
 {
  "name": "OfficialUSA",
  "url": "officialusa.com",
-  "version": "0.1.4",
+  "version": "0.1.5",


⚠️ The same here, we are not doing any changes to this broker

jotaemepereira

@Bunn Tests are not compiling on this PR

Bunn · 2024-02-28T20:02:07Z

@jotaemepereira I fixed the tests. Other than that, I believe that updating the version of the JSON files to take advancedbackgroundchecks.com changes into consideration is the only thing left right? The discussion about versioning is being held here

jotaemepereira

Looks good to me 👍🏼 . Thanks for addressing the comments

# Conflicts: # DuckDuckGo.xcodeproj/project.pbxproj # DuckDuckGo.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved # LocalPackages/DataBrokerProtection/Package.swift # LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Model/Actions/EmailConfirmation.swift # LocalPackages/NetworkProtectionMac/Package.swift # LocalPackages/SubscriptionUI/Package.swift

Bunn added 3 commits February 26, 2024 14:44

WIP: new datasource type

f6f36ca

Rename userData to userProfile

33284b3

Improve formatting

58dc407

github-actions bot added the bot: not in app board Added by automation for pull requests with tasks not added to macOS App Board Asana project label Feb 26, 2024

Bunn and others added 4 commits February 26, 2024 18:15

Update actions

7cb38ce

Add meta object to CCF response

f4f85b7

Update BSK to point to C-S-S 5.0

c5474bb

Update JSON

6af9d65

Bunn marked this pull request as ready for review February 28, 2024 14:49

Bunn removed the bot: not in app board Added by automation for pull requests with tasks not added to macOS App Board Asana project label Feb 28, 2024

Bunn requested a review from jotaemepereira February 28, 2024 15:20

brianhall mentioned this pull request Feb 28, 2024

Add February 2024 broker JSON #2273

Closed

github-actions bot added the bot: not in app board Added by automation for pull requests with tasks not added to macOS App Board Asana project label Feb 28, 2024

Change reference to bsk

10ed728

jotaemepereira reviewed Feb 28, 2024

View reviewed changes

jotaemepereira requested changes Feb 28, 2024

View reviewed changes

Fix tests

d8ac97a

Bunn requested a review from jotaemepereira February 28, 2024 20:43

Bump advancedbackgroundchecks.com.json

14f0fa0

jotaemepereira approved these changes Feb 28, 2024

View reviewed changes

Bunn added 3 commits February 29, 2024 11:07

Point to new BSK version

ea16eab

Update BSK to 113.0.0

dc0708c

github-actions bot removed the bot: not in app board Added by automation for pull requests with tasks not added to macOS App Board Asana project label Feb 29, 2024

Fix merge issue

f125094

Bunn merged commit abe67a7 into main Feb 29, 2024
17 checks passed

Bunn deleted the bunn/dbp/datasource branch February 29, 2024 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBP: Support new dataSource property #2267

DBP: Support new dataSource property #2267

Bunn commented Feb 26, 2024 •

edited

Loading

github-actions bot commented Feb 28, 2024 •

edited

Loading

jotaemepereira left a comment

jotaemepereira Feb 28, 2024

Bunn Feb 28, 2024

brianhall Feb 28, 2024

jotaemepereira Feb 28, 2024

Bunn Feb 28, 2024

brianhall Feb 28, 2024

Bunn Feb 28, 2024 •

edited

Loading

brianhall Feb 28, 2024

jotaemepereira Feb 28, 2024

Bunn Feb 28, 2024

brianhall Feb 28, 2024

jotaemepereira Feb 28, 2024

brianhall Feb 28, 2024 •

edited

Loading

jotaemepereira Feb 28, 2024

jotaemepereira left a comment

Bunn commented Feb 28, 2024

jotaemepereira left a comment

DBP: Support new dataSource property #2267

DBP: Support new dataSource property #2267

Conversation

Bunn commented Feb 26, 2024 • edited Loading

Internal references:

github-actions bot commented Feb 28, 2024 • edited Loading

jotaemepereira left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bunn Feb 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianhall Feb 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jotaemepereira left a comment

Choose a reason for hiding this comment

Bunn commented Feb 28, 2024

jotaemepereira left a comment

Choose a reason for hiding this comment

Bunn commented Feb 26, 2024 •

edited

Loading

github-actions bot commented Feb 28, 2024 •

edited

Loading

Bunn Feb 28, 2024 •

edited

Loading

brianhall Feb 28, 2024 •

edited

Loading